# Long context

ModernBERT-large is a state-of-the-art bidirectional encoder Transformer model (BERT style) pre-trained on 2 trillion tokens of English and code data, with a native context length of up to 8192 tokens. This model incorporates the latest architectural improvements such as Rotary Positional Embeddings (RoPE) for long-context support, local-global alternating attention for enhanced efficiency with long inputs, and padding-free and Flash Attention for improved inference speed. ModernBERT-long is suitable for tasks involving the handling of long documents, such as retrieval, classification, and semantic search within large corpora. The training data primarily consists of English and code, which may result in lower performance with other languages.

The AI21 Jamba 1.5 series of models is one of the most powerful long-context models available on the market, offering 2.5 times faster inference speeds than leading competitors. These models demonstrate exceptional long-context processing capabilities, speed, and quality, successfully scaling non-Transformer models to match the quality and strength of market-leading models for the first time.

Qwen1.5 is the next version in the Qwen series, offering an open-source version of the foundation and chat model that spans multiple scales from 0.5B to 72B. It supports multilingual capabilities, long context handling, aligns with human preferences, and achieves significant advancements in basic capabilities, chat performance, and multilingual understanding.

Flash-Decoding is a technique for long-context inference that can significantly accelerate the attention mechanism during inference, leading to an 8x improvement in generation speed. This technique achieves faster inference speed by parallelly loading keys and values and then rescaling and combining the results to maintain the correct attention output. Flash-Decoding is suitable for large language models and can handle long contexts such as long documents, long conversations, or entire codebases. Flash-Decoding is available in the FlashAttention package and xFormers, which can automatically select between Flash-Decoding and FlashAttention methods. It can also utilize the efficient Triton kernel.